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(54) TiUe: MOTION VECTOR PROCESSING 
(57) Abstract 

In a method of furnishing output motion vectors 
(MV*), motion vectors (MV) arc retrieved (DEC) from 
an encoded image signal (MPEG), and the retrieved 
motion vectors (MV) are post-processed to obtain 
the output motion vectors (MV*). The post processing 
includes generating (CVG) candidate motion vectors, 
and selecting (CVS) the output motion vectors (MV*) 
in dependence upon the candidate motion vectors and 
the retrieved motion vectors (MV). 
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The invention relates to a method and device for furnishing motion vectors, and 
to an image signal processing apparatus and an image display apparatus incorporating such a 
motion vector processing device. 



5 

DE- A- 195. 10.389 discloses a method and a circuit arrangement for receiving 
coded video signals. The method concerns a video signal which contains a sequrace of 
images with compressed redundancy, e.g. coded in accordance with the MPEG2-standard. 
The motion decoding is carried out first. The post-processing of the motion-decoded fields, 

10 which post-processing essentially contains interpolations, is carried out next together with the 
processing to remove flicker. Advantageously, the motion vectors retrieved at the motion 
decoding, are used also for the flicker removal. 

Motion estimation can be seen as an optimization problem: a sometimes 
complex, but usually simple criterion function has to be minimized or maximized to find an 

IS output motion vector. There are brute force methods that simply try all possible vectors, in a 
predefined range, in order to be sure to obtain the global optimum of the criterion function. 
Also there are efficient approaches that test only the most likely motion vectors. This 
likelihood is usually determined by ^tial or temporal proximity, and consequently temporal 
and spotisl prediction vectors have been pq>ular in the efficient motion estimation 

20 algorithms. Dq>ending on the motion estimation algorithm used, the properties of the 
resulting motion vector fields are different, so that the application areas of the various 
motion estimation algorithms are different. Motion vectors can be used in high quality video 
signal processing, like motion compensated scan ccmversion, motion compensated noise 
reduction and image coding. The intended processing puts constraints on various quality 

25 aspects of the motion vectors. For example, high quality motion compensated scan 

conversion requires a motion vector field where the vectors represent the true motion vectors 
(i.e., true direction and speed) within the image sequence. For coding ^plications this is 
irrelevant, as the only quality criterion is a low average prediction error. 

Brute force methods like full search block matching tend not to give true morion 
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vectors. Instead, fiiU search block matching mininiizes the error between blocks, so its 
vecton will connect two blocks that have a minimal error between these two blocks, which is 
suitable for minimizing the prediction error in coding applications. While these minimal 
errors work efficiently for compression algorithms like MPEG, the resulting vectors are 
5 unsuitable for use in high quality scan conversion. On the other hand, more efficient motion 
estimators like 3-D recursive block matching tend to give true motion vectors that can be 
used in high quality scan ccmversion but which may not be suitable for use in compression 
algorithms. 

With the emergence of MPEG and other types of video compression it is 

10 possible to have input video material that already contains motion vectors, so at first sight it 
may seem that for this type of source material motion estimation at the receiver side is not 
necessary. However, it is not certain that these motion vectors can be directly used for e.g. 
scan rate conversion, because it is imknown whether the MPEG modem vectors represent 
true motion or not. At the decoder/receiver side it is unknown what type of motion 

IS estimation algorithm was used at the encoder/transmitter side, so one must assume, as a 
worst case situation, that the MPEG nu>tion vectors are optimized for an efficient 
compression and that they do not rqnesent true motion vectors. For example, a 
homogeneous background could produce strange and long inconsistent vectors when small 
changes of brilliance happen in a heterogeneous way. More ^)ecifically, periodical structures 

20 and noise in picture areas with little detail, may cause such inconsistent vectors. Another 
problem with motion vectors in MPEG daU streams is that it is uncertain that all motion 
vectors are transmitted within the data stream. For these reasons, the technique proposed by 
DE-A-195. 10.389 seems not to be suitable for high quality post-processing. 

In an attempt to solve a sinular problem, it has been proposed to smooth the 

25 vectors transmitted with a digitally encoded television signal before using these vectors for a 
field rate upconversion operation, see the M.S.E.E. graduation report "Bewegings- 
gecompenseerde interpolatie van digitale telcvisid)eelden", by L.J. Steenbeek, Ddft 
University of Technology, December 1986, pp. 60-68. 

30 

It is, imer alia, an object of the invention to fiimish motion vecton which are 
suitable for high quality post-processing. To this end, a first aspect of the invention provides 
a method as defined by claim 1. A second aspect of the invention provides a device as 
defined by claim 4. A third zspcci of the invention provides an image signal processing 
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apparatus as defined by claim 5. A fourth aspect of the invention provides an image di^lay 
apparatus as defined by claim 6. Advantageous embodiments are defined in the dependent 
claims. 

In a method of furnishing output motion vectors in accordance with a primary 
S aspect of the present invention, motion vectors are retrieved from an encoded image signal, 
and the retrieved motion vectors are post*processed to obtain the ou^ut motion vectors. The 
post-processing includes generating candidate modem vectors, and selecting the output modon 
vectors in dependence upon the candidate modon vectors and the retrieved nK>don vectors. 

These and other aspects of the invention will be apparent from and elucidated 
10 with referoice to the embodiments described hereinafter. 

In the drawings: 

Fig. 1 illustrates in a schematic form various embodiments of an image display 
15 ^>paratus having a modon vector furnishing device in accordance with the present inventicm; 
and 

Fig. 2 discloses a H261/263 decoder with post-processing in accordance with 
the present invradon. 

20 

The problems described in the above background part do not exclude the MPEG 
modon vectors entirely from use in high quality scan conversion. When an appropriate post- 
processing is allied, the MPEG motion vectors can be made useful More specifically, 
when the receiver is able to determine the quality of the MPEG motion vectors, and when it 

25 is able to improve the quality of the MPEG motion vectors so that they meet certain criteria 
for the intended processing, the MPEG motion vectors can be used. Several possible post 
processing methods to extend the s^licability of the MPEG motion vectors are considered: 
1. Process the MPEG motion vectors with an operation like some sort of vector 

snooothing operation to increase the usability of the MPEG motion vectors. 

30 2. Use the MPEG motion vectors as additional candidates along with spatial and/or 

temporal candidates in, for instance, a 3*D recursive block matching algorithm, similar to the 
use of an additional parametric vector candidate as disclosed in the non-prepublished EP 
patent application no. 96201462.7 filed on 24.05.96 and in any corresponding patent 
s^lications, incorporated by reference herein (attorney's docket PHN 15,824). For a 
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description of an efficient recursive nru>tion vector estimation algorithm which obtains motion 
vectors on the basis of several candidate vectors, reference is also made to the articles by 
Dr. G. de Haan et al. listed at the end of the description of the non-prepublished EP patent 
application no. 96201462.7. 
5 3. The MPEG moticm vectors can be used to calculate the parametric vector 

candidates described in the non-prq)ublished EP patent application no. 96201462.7. 

For a schemadc description of the various embodiments, reference is made to 

Fig. 1. 



Fig. 1 shows a basic section B, and four sections El .. E4 showing four distinct 
ways to augment the ba$ic section B to obtain four distinct embodiments B+El, B+E2, 
B+E3, and B+E3+E4 of an image di^iay apparatus. 

In the basic section B, an MPEG signal comprising a video signal V and motion 
vectors MV, is allied to a decoder DEC in order to retrieve the video signal V in 
dependence upon the motion vectors MV from the MPEG input signal. The video signal V is 
applied to a video post-processor VPP to obtain an ouq>ut video signal OVS having a 
doubled field frequency and/or line frequency or an improved motion portrayal with respect 
to the video signal V. Alternatively or in addition, the video post-processor VPP reduces 
noise and/or changes the number of pixels per line and/or lines per field of the video signal 
V. The video post-processing carried out by the processor VPP is dependent on improved 
^-->-^mmmwmSiS^V/^^ display device D. 

In accordance with a first embodiment B+El, the improved motion vectors 
MV' are obtained from the motion vectors MV contained in the MPEG signal by means of a 
post-processor PP. For example, a smoothing operation will reduce most of the artifacts 
25 which would arise if the MPEG motim vectors MV were directly used for the video post- 
processing carried out by the processor VPP. A smoothing operation may include separate 
low-pass filtering operaticms on the horizontal and vertical components of the motion vecton. 
Alternatively, the horizontal and vertical components of the motion vectors can be applied to 
median filters to rq>lace extreme motion vector component values by existing motion vfctor 
30 component values of motion vectors of neighboring blocks of picture elements. 

In accordance with a second embodiment B+E2 which is covered by the claims 
of the present patent ^plication, the MPEG motion vectors MV are multiplexed by a 
multiplexer MUX with candidate vectors generated by a candidate vector generator CVG, 
optionally after a (e.g. smoothing) post-processing operation carried out by the post- 
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processor PP. The multiplexed vectors are applied to a candidate vector selector CVS to 
obtain improved motion vectors MV' which are expected to show the smallest errors. 

In a third embodiment B+E3 which is covered by a copending patent 
sq>plicaticHi, a parameter extractor PE extracu modon parameters pi pn from the MPEG 
5 motion vectors MV. The moticm parameters pi pn are applied to a vector field calculator 
VFC to obtain the improved motion vectors MV\ Reference is again made to the non- 
prepublished EP patent applicaticm no. 96201462.7 and its corresponding ^plications for 
further details. 

In a fourth embodiment B+E3+E4 which is covered by the claims of the 
10 present patent application, the modcMi vectors calculated by the vector field calculator in 

secdon £3 are multiplexed with candidate modon vectors from the candidate vector goierator 
CVG, and ^lied to the candidate vector selector CVS to obtain the improved motion 
vectors MV\ This fourth embodiment combines the features of the second and third 
embodiments. 

15 

Obviously, the present invention is not limited to MPEG signals; other signals 
like H261 or H263 signals can be treated in the same manner. Fig. 2 discloses a H261/263 
decoder with post-processing in accordance with the present invmtion. A H261 or H263 
input signal is applied to a demultiplexer DEMUX to obtain a digital video signal DV and 

20 modon vectors MVE. The motion vectors MVE are applied to a variable length decoder 
VLC^ to obtain decoded motion vectors MV. The digital video signal DV is applied to a 
cascade connection of a variable length decoder VLC*, an inverse quantizer Q'\ and a 
discrete cosine transform (DCT) decoder DCT\ to obtain a video difference signal VDS. 
The video difference signal VDS and the motion vectors MV are ^>plied to a prediction 

25 block PR, whose ou^ut is added to the video difference signal VDS to obtain the decoded 
video signal V. The decoded video signal V and the motion vectors MV are allied to a 
recursive motion estimator RME to obtain improved motion vectors MV* in dependence on 
which the decoded video signal V is processed by a motion-compensated interpolator MCI to 
obtain an ou^ut video signal OVS. In a sequence of coded images comprising intra-frame 

30 coded images (which stand alone), predictively coded images (which can be decoded by 
means of a preceding intia-frame decoded image or a preceding decoded predictively coded 
image, and the motion vectors MV), and bi-directionally coded images between an intra- 
fiame coded image and a predictivdy coded image or between two predictively coded 
images, the improved motion vectors MV' are preferably used to decode (i.e. interpolate) the 
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bi-directionally coded images. 

The embodiment of Fig. 2 can be implemented on a PC add-on card in a multi- 
media PC, 



5 The basic considerations underlying a primary aqject of the present invention 

can be summarized as follows. Different motion estimation algorithms will have different 
applications based on the properties of their resulting vector fields. NfPEG video data streams 
contain motion vectors, which can possibly be used in high quality video processing. 
However, it is likely that these MPEG motion vectors are c^timized to achieve an efficient 

10 compression, so that these vectors cannot be expected to represent the true motion. This 
means that additional processing is needed to improve their quality and usability for a high 
quality post-processing. It is proposed to process the MPEG motion vectors at the receiver 
side. For instance, they can be used as additional candidate vectors in an efficient motion 
estimator. In this manner, the a^licability of MPEG motion vectors is extended. 

15 The following salient features of preferred embodiments of the present invention 

can be menticHied. A post-<^)eiation on motion vectors, that are received along with a video 
signal, intended to further increase a quality aspect of said vectors like e.g. their q)atial 
and/or temporal smoothness, so that they can be used for high quality video signal 
proces^g. A post operation on motion vectors, that are received along with a video signal, 

20 where the motion vectors are used, possibly after a post processing, as additional candidates 

possible vector caiuiidates. An operation where the motion vectors are used the generate the 
parameters of a parametric model that describes the global motion within the video sequence. 
An operation where these extracted parameters are used to generate additional candidates for 
25 some, or all, of the blocks in an efficient motion estimator that uses a limited set of possible 
vector candidates. 

It should be noted that the above-mentioned embodiments illustrate rather than 
limit the invention, and that those skilled in the art will be able to design many alternative 
embodiments without departing from the scope of the appended claims. In the claims, any 
30 reference signs placed between parwitheses shall not be construed as limiting the claim. The 
invention can be implemented by means of hardware comprising several distinct elements, 
and by means of a suitably programmed computer. Reference is made to copending patent 
applications for features disclosed but not claimed in this application. 
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1. A method of fumishuig output motion vectors (MV'), the method comprising 
the steps of: 

retrieving (DEC; DEMUX, VLC*) motion vectors (MV) from an encoded 
image signal (MPEG; H261, H263); and 
S post-processing (RME) the retrieved motion vectors (MV) to obtain the output 

motion vectors (MV*), wherein the post-processing step (RME) includes the steps of: 

generating (CVG) candidate motion vectors; and 

selecting (CVS) said output moticm vectors (MV*) in dq)endence upon said 
candidate motion vectors and said retrieved motion vectors (MV). 

10 

2. A method as claimed in claim 1, wherein the post-processing step (RME) 
includes the step of subjecting said retrieved motion vectors (MV) to a smoothing operation 
(PP) to obtain smoothed vectors, and where in said selecting step (CVS), said output motion 
vectors (MV*) are selected horn said candidate motion vectors and said smoothed vectors. 

15 

3. A method as claimed in claim 1, wherein the post-processing step (RME) 
includes the steps of: 

extracting (PE) motion parameters (pi pn) from said retrieved motion vectors 

(MV); 

20 calculating parametric motion vectors in dependence upon said moticm 

parameters (pi pn); and 

where in said selecting st^ (CVS), said output motion vectors (MV') are 
selected from said candidate motion vectors and said parametric nK>tion vectors. 
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4. A device for furnishing output motion vectors (MV*), the device comprising: 

means for retrieving (DEC; DEMUX, VLC*) motion vectors (MV) from an 

encoded image signal (MPEG; H261, H263); and 

means for post-processing (RME) the retrieved motion vectors (MV) to obtain 
5 the ouq>ut motion vectors (MV), wherein the post-processing means (RME) include: 
means for generating (CVG) candidate motion vectors; and 
means for selecting (CVS) said output motion vectors (MV*) in dependence 

upon said candidate motion vectors and said retrieved motion vectors (MV). 

10 5. An image signal processing ^paratus, comprising: 

an output motion vector furnishing device (DEC, PP; DEMUX, VLC S RME) 
as claimed in claim 4; 

means (DEC; VLCS Q *, DCT', PR) for providing a decoded image signal (V) 
in response to said encoded image signal (MPEG; H261, H263); and 
15 means for post-processing (VPP; MO) said decoded image signal (V) in 

dependence upon said ouq>ut motion vectors (MV*), to obtain a post-processed image signal 
(OVS). 



6. 

20 



An image display apparatus, comprising: 

an image signal processing s^paratus as claimed in claim S; and 

means (D> for di^laying said post-prt»cessed image sigtial (OVS). 
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(57) Abstract 

In a method of furnishing output motion vectors 
(MV), motion vectors (MV) arc retrieved (DEC) from 
an encoded image signal (MPEG), and the retrieved 
motion vectors (MV) are post-processed to obtain 
the output motion vectors (MV). The post-processing 
includes generating (CVG) candidate motion vectors, 
and selecting (CVS) the output motion vectors (MV) 
in dependence upon the candidate motion vectors and 
the retrieved motion vectors (MV). 
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